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Abstract 
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cost-effective  manner.  In  order  to  accomplish  its  mission,  MPT&E  strives  to  provide 
the  right  sailor  with  the  right  skill  sets  in  the  right  job  at  the  right  time,  and  to  manage 
his/her  career  path  in  support  of  warfighting  capabilities.  To  support  this  objective, 
some  form  of  automated  learning  management  system  or  virtual  learning 
environment  is  needed.  A  learning  management  system  (or  LMS)  is  a  software 
application  that  enables  the  delivery  and  management  of  online  content  to  learners. 
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Executive  Summary 


A  key  element  of  the  Navy’s  Manpower,  Personnel,  Training  and  Education 
(MPT&E)  mission  is  to  recruit,  develop,  manage  and  deploy  a  workforce  in  an  agile, 
cost-effective  manner.  In  order  to  accomplish  its  mission,  MPT&E  strives  to  provide  the 
right  sailor  with  the  right  skill  sets  in  the  right  job  at  the  right  time,  and  to  manage  his/her 
career  path  in  support  of  warfighting  capabilities.  To  support  this  objective,  some  form  of 
automated  learning  management  system  or  virtual  learning  environment  is  needed.  A 
learning  management  system  (or  LMS)  is  a  software  application  that  enables  the 
delivery  and  management  of  online  content  to  learners. 

This  research  investigates  the  feasibility,  suitability,  and  use  of  learning 
management  systems  to  support  online  and  blended  learning  in  the  Navy.  The  research 
addresses  many  issues  related  to  the  selection,  implementation,  and  use  of  learning 
management  systems,  such  as:  the  current  state  of  the  learning  management  systems 
industry,  the  essential  functional  and  technical  requirements  of  an  effective  learning 
management  system,  the  process  and  pitfalls  for  selecting,  implementing,  and 
evaluating  a  learning  management  system,  the  case  against  using  learning 
management  systems,  and  future  trends  and  directions  of  LMS  applications. 

A  major  benefit  of  using  a  learning  management  system  is  to  facilitate  “anytime, 
any  place,  any  pace”  access  to  learning  content  and  administration.  Typically,  a  LMS 
allows  for  learner  registration,  delivery  of  learning  activities,  learner  assessment  and 
tracking,  and  report  generation  on  learner  progress  and  assessment  results.  Advanced 
LMS  features  include  competency  management,  skills-gap  analysis,  succession 
planning,  certifications,  and  resource  allocation.  As  such,  a  LMS  should  be 
considered  a  high-level,  mission-critical,  strategic  solution  for  planning, 
delivering,  and  managing  all  learning  events  within  MPT&E — including  online, 
virtual  classroom,  and  instructor-led  courses. 

Although  LMSs  are  extremely  useful  in  delivering  and  managing  structured 
content,  if  learning  is  based  on  discovery,  exploration,  and  thinking,  the  artificial  and 
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structured  approach  of  content  delivery  and  interaction  of  a  LMS  could  be  very  limiting. 
In  this  situation,  rather  than  a  adopting  a  large,  centralized,  closed,  mono-cultural  LMS 
that  dictates  the  nature  of  interaction  between  learners,  instructors,  and  content,  we 
recommend  that  the  Navy  consider  adopting  an  integrated  learning  environment 
(ILE)  that  consists  of  a  set  of  diverse  tools  that  provide  students  with  connected 
specialization,  modularization,  and  decentralization.  Such  an  environment  would 
place  the  learner  at  the  center  of  learning  and  would  allow  him/her  to  explore  various 
areas  and  directions  of  personal  interest.  It  would  integrate  blogs,  wikis,  social 
networking  tools,  collaborative  spaces,  knowledge  management,  “connecting-enabling” 
protocols  like  RSS,  and  other  tools.  The  intent  of  the  ILE  is  to  give  the  learner  the 
control  needed  to  attain  his  or  her  own  personal  learning  goals. 

Regardless  of  which  approach  is  adopted,  a  crucial  first  step  for  the  Navy  is  to 
have  a  clear  understanding  of  the  virtual  learning  environment  and  develop  a 
clear  learning  strategy.  A  learning  strategy  should  include,  among  other  elements,  the 
nature  of  the  learning  to  be  carried  out,  the  characteristics  of  the  learners,  how  learning 
is  to  be  delivered  to  them,  what  learning  pedagogy  is  appropriate  for  them,  the  optimal 
level  of  social  interaction  for  learning,  and  available  resources.  Organizational  goals  and 
objectives  should  also  be  defined  clearly  and  the  strategy  aligned  to  them. 

This  research  also  supports  the  need  to  develop  a  learning  architecture  as 
a  prerequisite  to  implementing  a  learning  management  system  or  an  integrated 
learning  environment.  A  learning  architecture  defines  the  basic  functionality  required 
from  a  learning  management  system  or  integrated  learning  environment,  such  as: 
learner  registration,  delivery  of  learning  activities,  learner  assessment,  tracking,  and 
reporting.  It  should  also  include  advanced  functionality,  such  as:  content  authoring, 
competency  management,  skills-gap  analysis,  succession  planning,  certifications,  and 
resource  allocation.  A  learning  architecture  also  defines  links  to  external  information 
sources  and  external  communities. 

It  is  easy,  yet  dangerous,  to  let  technology  determine  how  learning  occurs. 
Therefore,  before  selecting  a  technology  solution,  it  is  crucial  that  Navy  decision-makers 
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clearly  determine  and  prioritize  requirements  in  order  to  find  the  most  suitable  learning 
system  to  meet  the  Navy’s  current  mission  requirements,  as  well  as  one  which  can  grow 
to  meet  subsequent  and  changing  future  requirements. 
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I 


Introduction 


Online  and  blended  delivery  of  training  and  professional  development  is  an 
important  feature  of  the  contemporary  workplace.  A  wide  range  of  corporate,  public 
and  educational  organizations  use  Information  and  Communication  Technologies 
(ICT)  to  support  the  education  of  their  workforce.  E-learning  allows  learners  to 
access  learning  without  the  expense  and  disruption  involved  in  attending  courses  at 
a  particular  geographic  location. 

To  implement  e-learning  successfully,  an  organization  requires  some  form  of 
learning  management  system  or  virtual  learning  environment  to  support  the  learning 
process.  Learning  management  systems  (or  LMSs)  are  software  packages  that 
enable  the  delivery  and  management  of  online  content  to  learners.  Most  LMSs  are 
web-based  to  facilitate  "anytime,  any  place,  any  pace"  access  to  learning  content 
and  administration  ("Learning  management  systems,”  2007).  Typically,  a  LMS  allows 
for  learner  registration,  delivery  of  learning  activities,  learner  assessment  and 
tracking  in  an  online  environment.  A  key  component  of  a  LMS  is  the  ability  for 
managers  to  generate  reports  on  learner  progress  and  assessment  results.  More 
comprehensive  LMSs  often  include  tools  such  as  competency  management,  skills- 
gap  analysis,  succession  planning,  certifications,  and  resource  allocation.  As  such,  a 
LMS  can  be  considered  a  high-level,  strategic  solution  for  planning,  delivering,  and 
managing  all  learning  events  within  an  organization — including  online,  virtual- 
classroom,  and  instructor-led  courses. 

There  is  a  plethora  of  such  systems  available,  ranging  from  commercial 
offerings  such  as  Sumtotal,  Saba,  and  Plateau  Systems  to  open  source  solutions 
such  as  Moodle,  First  Class,  and  Interact.  The  question  of  which  software  to  use  for 
a  given  learning  situation  is  complex.  Factors  to  be  considered  include  the  nature  of 
the  learning  to  be  carried  out,  the  characteristics  of  the  learner,  the  preferred 
pedagogy,  IT  support  and  cost. 
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1. 


Research  Questions 

The  aim  of  the  current  research  was  to  answer  the  following  research 
questions. 

1 .  What  is  the  current  state  of  the  learning  management  systems 
industry? 

2.  What  are  the  essential  functional  and  technical  requirements  of 
learning  management  systems  to  make  them  effective  learning  tools  in 
the  workplace? 

3.  What  is  the  process  for  selecting  and  implementing  a  learning 
management  system? 

4.  What  are  the  pitfalls  for  selecting  and  implementing  a  learning 
management  system? 

5.  How  can  learning  using  a  learning  management  system  be  evaluated? 

6.  What  is  the  case  against  using  a  learning  management  system  in  an 
organization? 

7.  What  does  an  ideal  learning  environment  for  effective  organizational 
learning  look  like? 

8.  What  are  the  trends  for  learning  management  systems  for  2008  and 
beyond? 
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II 


Research  Methodology 


The  present  research  comprises  a  review  of  the  literature  comparing  various 
online  learning  environments  and  a  detailed  study  of  the  experiences  of  users  and 
developers  of  these  environments. 

1.  Data  Sources 

Numerous  data  sources  were  used  for  this  research.  Most  of  the  literature 
reviewed  was  obtained  from  the  Internet  via  Google  and  Google  Scholar  searches. 
The  material  published  online  was  supplemented  by  a  number  of  Master’s  theses 
and  PhD  dissertations  related  to  the  topic  of  research. 

In  addition,  a  number  of  informal  interviews  and  discussions  were  carried  out 
at  academic  and  corporate  conferences  with  a  range  of  people  who  interact  with 
LMSs,  such  as  learners,  instructors,  administrators  and  vendors.  Based  on  the 
research  questions,  different  sets  of  guideline  questions  were  generated  for  the 
different  categories  of  users.  While  the  guideline  questions  were  used  as  a  focus  for 
interviews,  actual  interviews  tended  to  be  fairly  wide-ranging  around  these 
questions. 

2.  Data  Analysis 

A  comparative  analysis  was  used  to  evaluate  the  data  collected  in  order  to 
answer  the  research  questions.  Interview  recordings  were  transcribed  and  compared 
to  the  written  interview  notes.  Common  themes  from  different  interviews  were 
combined  to  gain  perspective  on  each  of  the  research  questions.  A  similar  approach 
was  taken  to  analyzing  written  material  obtained  from  Internet  sources  and 
published  articles. 
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III.  The  Learning  Management  Systems  Industry 


The  birth  and  growth  of  the  Internet  in  the  early  1990s  ushered  a  new  era  of 
learning  delivery.  It  was  during  the  Internet  boom  of  the  mid-  to  late-1990s  that 
computer  managed  instruction  (CMI)  systems  used  to  manage  computer-based 
training  (CBT)  evolved  into  learning  management  systems  (LMS).  As  such,  the  LMS 
is  considered  a  relatively  new  type  of  software  application. 

Although  the  core  functionality  of  a  LMS  is  to  register  learners,  provide 
content  to  learners,  track  courses  in  a  catalog,  record  data  from  learners,  and 
provide  appropriate  reports  to  management,  “LMS”  means  different  things  to 
different  people.  The  functionality  and  feature  list  of  each  LMS  is  often  rich  with 
descriptions  that  can  be  interpreted  in  numerous  ways.  To  further  complicate  the 
landscape,  every  LMS  offered  by  vendors  boasts  specialized  features  in  an  attempt 
to  provide  that  vendor  with  a  competitive  edge. 

There  is  a  surprisingly  large  vendor  base  of  LMSs.  In  2000,  there  were  well 
over  100  learning  management  systems  (LMS)  on  the  market  (Hall,  2001).  In  the 
subsequent  years,  these  figures  have  dropped  slightly  due  to  consolidation,  mergers 
and  acquisitions.  In  addition  to  LMSs,  other  learning  systems  offering  overlap 
functionality  include:  Learning  Content  Management  Systems  (LCMS),  Knowledge 
Management  Systems  (KMS),  Performance  Management  Systems  (PMS),  Talent 
Management  Systems  and  many  combinations  of  the  above.  Over  the  past  3  years, 
numerous  LMS  vendors  began  to  either  add  or  merge  with  other  systems  to  create  a 
more  integrated  learning  management  system.  The  LMS  market  appears  to  be  in  a 
continuous  state  of  flux. 

It  is  important  to  note  that  many  LMS  vendors  are,  in  reality,  content  providers 
who  develop  software  to  manage  the  content  they  provide  rather  than  manage  a  full 
corporate  education  or  training  program. 
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IV.  Functional  and  Technical  Requirements  of  a 
LMS 

One  of  the  biggest  mistakes  organizations  make  is  to  let  technology  drive  the 
selection  of  a  learning  management  system.  Therefore,  it  is  important  for 
organizations  to  clearly  define  and  prioritize  requirements  before  looking  for  a  LMS. 
Once  requirements  are  identified,  they  can  then  be  matched  to  product  functionality. 
In  the  following  sections,  we  discuss  essential  functional  and  technical  requirements 
of  a  LMS: 

1.  LMS  Functional  Requirements 

a.  Learner  Features 

To  best  serve  learners,  a  LMS  must  allow  them  to  do  the  following  quickly 
and  easily  (elementk,  2003;  Learning  Circuits,  2005): 

■  Enroll  and  keep  track  of  their  progress  in  various  curricula,  courses, 
and  other  learning  events, 

■  Assess  their  skills  by  identifying  their  competency  gaps,  matching 
those  gaps  to  a  prescribed  curriculum  to  fill  those  gaps,  and 
developing  a  learning  plan  to  close  those  gaps  that  were  identified, 

■  Access  content  using  different  media  (e.g.,  classroom,  CD-ROM, 
online,  etc.),  different  methods  (e.g.,  instructor-led,  self-paced, 
blended),  and  different  languages, 

■  Complete  assessments  prior  to  starting  courses  so  they  can  target 
learning  time  most  effectively,  and 

■  Access  reference  materials  to  supplement  courses. 

b.  Administrator  Features 

A  good  LMS  should  deliver  five  critical  managerial  functions  to  administrators: 
learner  management,  content  assignment,  tracking  and  reporting,  content 
development,  and  communication  (elementk,  2003;  Learning  Circuits,  2005).  The 
following  is  a  closer  look  at  each  of  these  functions: 
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Learner  Management 


o  Create  and  issue  access  to  the  system, 

o  Manage  user  registrations  and  profiles, 

o  Manage  all  class-related  resources,  including  classrooms  and 
instructors, 

o  Organize  administrators  and/or  students  into  groups  for 
reporting  and  content-assignment  purposes,  and 

o  Integrate  appropriate  support  tools — including  exercises, 
reference  materials,  labs,  tests,  and  collaboration  tools. 

Content  Assignment 

o  Organize  courses  and  events  in  catalogs  that  are  intuitive  and 
searchable, 

o  Target  content  to  the  correct  individuals  or  groups, 

o  Designate  selected  content  as  “required”  learning;  allow 
students  to  select  other  courses  as  they  desire, 

o  Create,  edit,  distribute,  and  deliver  assessments, 

o  Develop  certification  and  personalized  learning  paths, 

o  Include  learners’  job  roles  and  functions,  and 

o  Deliver  online,  instructor-led  courses  in  synchronous  or 

asynchronous  modes — including  course  setup,  syllabus  display, 
and  registration  and  tracking. 

Tracking  and  Reporting 

o  Track  and  report  on  student  progress  and  activity, 

o  Track  and  report  professional  development  progress  against  a 
predefined  set  of  training  goals, 

o  Track  and  report  on  compliance  training  deployment,  and 
o  Verify  whether  knowledge  has  been  retained  after  training. 

Content  Development 
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o  Develop  new  proprietary  courses, 

o  Develop  new  courses  through  a  third  party, 

o  Use  templates  to  speed  development,  and 

o  Integrate  proprietary  courses  seamlessly  into  the  learning 
environment. 

■  Communication 

o  Enable  communication  between  administrators  and  learners, 

o  Search  and  identify  learners  and  deliver  targeted  courses, 

news,  references,  and  other  information  to  continually  engage 
them,  and 

o  Supplement  courses  by  integrating  specific  learning  references 
into  the  platform  and  by  allowing  learners  access  to  external 
resources. 

c.  Enterprise  Features 

In  addition  to  Learner  and  Administrator  features,  a  good  LMS  platform 
should  provide  methods  to  closely  monitor  whether  the  right  employees  are  taking 
the  right  courses — and  whether  they’re  getting  the  information  they  need  from  them 
(elementk,  2003).  It  should  be  able  to  identify  employees  who  need  a  particular 
course  and  inform  them  how  it  fits  into  their  overall  career  path,  when  it  is  available, 
how  it  is  available,  if  there  are  prerequisites,  and  when  and  how  they  can  fulfill  those 
prerequisites. 

Once  a  learner  completes  a  course,  the  LMS  can  administer  tests  based  on 
proficiency  requirements,  report  test  results,  and  recommend  next  steps.  As  such, 
LMSs  are  critical  in  assuring  that  enterprises  meet  compliance  and  certification 
requirements. 

2.  LMS  Technical  Requirements 

In  the  following  section,  we  examine  critical  technical  requirements  of  a  LMS 
and  why  they  are  critical  to  the  function  of  any  enterprise  management  system 
(elementk,  2003;  Hall,  2003). 
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High  availability:  The  LMS  must  be  able  to  handle  the  different  needs 
of  thousands  of  learners,  instructors,  administrators,  and  content 
builders  simultaneously. 

Scalability:  The  system  should  be  able  to  expand — or  “scale” — to  meet 
increasing  demands  (both  in  terms  of  the  volume  of  instruction  and  the 
size  of  the  learning  body)  without  compromising  performance. 

Extensibility:  The  system  should  be  able  to  easily  add  new  functions 
and  features  as  they  become  available — for  example,  connectivity  and 
integration  with  other  enterprise  systems. 

Usability:  The  access,  delivery  and  presentation  of  material  must  be 
easy-to-use  and  highly  intuitive. 

Configurability:  The  system  should  be  able  to  configure  the  user 
interface,  administrator’s  reports,  and  other  parameters  to  meet 
requirements. 

Interoperability:  The  LMS  should  be  able  to  support  content  from 
different  sources  and  multiple  vendors’  hardware/software  solutions. 

Support  of  Industry  Standards:  The  LMS  should  be  based  on  open 
industry  standards  for  Web  deployments  (XML,  SOAP  or  AQ)  and 
support  the  major  learning  standards  (AICC,  SCORM,  IMS  and  IEEE). 

Stability:  The  LMS  infrastructure  should  be  able  to  reliably  and 
effectively  manage  a  large  enterprise  implementation  running  around 
the  clock. 

Security:  The  LMS  should  be  able  to  selectively  limit  and  control 
access  to  online  content,  resources  and  back-end  functions,  both 
internally  and  externally.  Security  measures  usually  include  user 
IDs/passwords  and  encryption. 


V.  Process  for  Selecting  and  Implementing  a  LMS 


The  following  seven-step  process  adapted  from  Alvarado  (2004)  and 
Learning  Circuits  (2005)  should  be  considered  as  a  general  guideline  for  the 
selection  process. 

1.  Understand  the  Learning  Environment  and  Determine  a 
Learning  Strategy 

The  fundamental  first  steps  in  selecting  a  LMS  solution  are  to  understand  the 
learning  environment  and  to  develop  a  clear  learning  strategy.  In  developing  a 
strategy,  the  following  should  be  considered:  The  nature  of  the  learning  to  be  carried 
out  (e.g.,  simple  assimilation  of  facts  and  information — such  as  safety  or  operating 
procedures — or  deeper  personal  development  such  as  occurs  in  management  or 
leadership  programs),  the  characteristics  of  the  learners  (e.g.,  age,  gender, 
educational  background,  computer  literacy  and  accessibility,  geographical  location), 
the  number  of  learners,  what  learning  pedagogy  is  appropriate  for  the  learners  (e.g., 
transmissive  or  learner-centered),  the  level  of  social  interaction  for  learning,  the  level 
of  IT  support  within  the  organization,  budget  constraints  (time,  resources,  money), 
available  software  that  appears  appropriate,  and  installation  and  maintenance  issues 
(e.g.,  hardware  platforms,  skills  available,  server  space,  available  user  support). 
Corporate  goals  and  objectives  should  also  be  defined  and  the  learning  strategy 
aligned  to  them.  Critical  success  factors  should  be  indentified  and  given  a  highest 
priority.  Potential  realized  benefits  and  return  on  investment  should  also  be  taken 
into  consideration. 

Development  of  a  learning  strategy  is  a  very  crucial  and  non-trivial  task. 
Indeed,  it  can  be  a  separate  initiative  in  itself.  A  learning  strategy  should  reflect  how 
learning  programs  are  delivered  to  the  people  who  need  them  to  accomplish 
business  goals. 
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2.  Identify  and  Document  Requirements 

Specific  requirements  should  be  defined  in  each  of  the  areas  mentioned 
previously.  One  of  the  key  factors  in  finding  the  right  LMS  for  an  organization  is 
matching  a  LMS  to  requirements,  not  the  other  way  around. 

It  is  also  important  to  prioritize  requirements  in  a  range  from  “must  have”  to 
“nice  to  have.”  “Must  have”  requirements,  or  core  requirements,  are  those  which  the 
LMS  must  meet  within  the  initial  implementation  of  the  system.  Any  LMS  that  does 
not  meet  a  “must  have”  requirement  should  be  dismissed  from  further  consideration. 
“Important  to  have”  requirements  are  essential  in  the  initial  or  subsequent  phases  of 
implementation.  This  means  that  the  LMS  may  not  be  able  to  meet  an  “important  to 
have”  requirement  for  the  initial  phase,  but  a  new,  scheduled  release  appears  to 
meet  the  requirement  or  there  is  a  commitment  from  the  LMS  vendor  to  meet  the 
requirement  in  the  near  future.  Finally,  “nice  to  have”  requirements  can  be  delayed 
indefinitely,  but  could  be  promoted  to  “important  to  have”  requirements,  so  specifying 
“nice  to  have”  requirements  still  have  a  bearing  on  how  willing  or  open  a  LMS  vendor 
is  to  considering  them. 

Another  consideration  is  how  the  gaps  between  the  requirements  and  the 
capabilities  of  a  particular  LMS  can  be  filled.  Usually,  this  is  accomplished  through 
configuration  or  customization  of  the  LMS.  Configuration  refers  to  changeable 
parameters  within  the  limitations  of  the  code  base  of  the  LMS,  while  customization 
refers  to  the  ability  to  add  additional  functionality  (such  as  integration  with  other 
systems)  not  originally  included  in  the  LMS  design.  Customization,  however,  works 
against  the  flexibility,  scalability,  and  efficiency  of  the  system.  Since  customization  is 
hard-coded,  it  usually  requires  extensive  programming  every  time  a  new  version  of 
the  LMS  is  used  or  when  business  conditions  change. 

Another  important  consideration  is  the  hosted  versus  in-house  installed 
solution.  Hosted  systems  are  maintained  by  the  LMS  provider,  which  acts  as  an 
application  service  provider  (ASP).  The  LMS  provider  typically  grants  access  to 
users  of  the  LMS  and  provides  support  for  the  system  should  problems  arise. 
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Modifications  or  customizations  beyond  what  the  application  supports  in 
configuration  screens  may  need  to  be  done  by  the  ASP  and  can  be  restrictive. 
Installed  solutions,  on  the  other  hand,  are  systems  that  are  installed  on  the 
company’s  servers  and  network.  The  support  of  the  hardware  and  applications 
would  most  likely  fall  on  the  IT  organization,  but  there  would  be  greater  control  over 
configuration  and  customizations  of  the  LMS  than  in  a  hosted  solution.  Compatibility 
to  standards  such  as  SCORM  (Shareable  Content  Object  Reference  Model)  and 
AICC  (Aviation  Industry  CBT  Committee)  should  also  be  considered. 

3.  Research  LMS  Companies 

Following  the  identification  and  documentation  of  requirements,  it  will  be 
necessary  to  research  potential  LMS  vendors.  Information  on  each  vendor  is  usually 
available  on  its  respective  Web  site.  Additionally,  research  and  comparison  reports 
may  also  be  available  from  industry  research  firms  such  as  Gartner,  IDC,  and  the 
META  Group.  It  is  also  useful  to  talk  to  customers  in  the  same  industry  about  their 
experiences  with  the  vendors  of  their  existing  LMSs. 

When  researching  LMS  companies,  decision-makers  should  focus  on  key 
areas  surrounding  the  core  or  “must  have”  requirements.  LMS  companies  will 
usually  work  with  a  client  to  fulfill  “must  have”  requirements  through  partnerships, 
customization  or  future  releases.  From  these  reports,  a  short  list  of  companies  to 
which  requests  for  proposals  can  be  sent  should  emerge. 

4.  Prepare  the  Request  for  Proposal  (RFP) 

Based  on  the  learning  strategy  and  requirements  identified  in  Steps  1  and  2 
above,  a  Request  for  Proposal  (RFP)  should  be  prepared  and  sent  to  the  selected 
potential  vendors.  A  RFP  is  a  written  document  that  outlines  specific  requirements 
suppliers  must  meet  in  order  to  win  the  buyer’s  business.  In  the  RFP,  it  is  not 
necessary  for  organizations  to  indicate  priorities  of  requirements,  nor  list  them  in  any 
specific  order;  thus,  each  requirement  is  responded  to  equally.  Each  requirement 
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should  be  as  specific  as  possible  so  that  the  LMS  vendor  can  respond  directly  to  the 
requirement  rather  than  provide  a  general  response. 

Use  cases  or  scenarios  should  also  be  included  in  the  RFP.  Use  cases 
describe  very  specific  situations  important  to  the  organization  that  the  LMS  needs  to 
accommodate.  These  will  give  the  LMS  vendor  a  clear  indication  as  to  how  it  can 
meet  specific  situations. 

The  RFP  should  also  include  a  request  for  a  proposed  project  plan  for 
implementation  based  on  the  requirements.  The  project  plan  must  include  timelines 
of  the  different  project  tasks.  This  will  provide  an  estimate  as  to  how  long  the  vendor 
perceives  implementation  will  take,  the  ownership  for  each  task,  and  the  details  of 
the  tasks  themselves.  Usually,  experienced  LMS  vendors  would  already  have  a 
template  of  a  project  plan  that  could  easily  be  applied  in  a  proposal. 

Finally,  it  is  important  to  request  a  short  response  time  for  the  RFP.  This  will 
give  an  indication  as  to  how  hard  a  company  will  work  for  the  business  and  can  be  a 
strong,  but  not  the  only,  indicator  as  to  how  it  will  perform  in  a  business  relationship. 

5.  Review  the  Proposals 

A  representative  review  committee  should  review  the  proposals  and  establish 
an  agreed-upon  rating  system.  Each  proposal  should  be  rated  according  to  the 
rating  system.  Decision-makers  should  also  include  subjective  comments  for  both 
positive  and  negative  impressions  in  the  rating  documentation.  This  allows 
subjective  impressions,  in  addition  to  the  quantitative  measures  of  each  criterion,  to 
be  incorporated  into  the  process. 

The  goal  of  the  proposal  review  is  to  eliminate  rather  than  to  select.  Hence, 
the  focus  of  the  proposal  review  should  be  on  the  core,  or  “must  have,” 
requirements.  These  are  usually  the  requirements  that  must  be  present  in  order  for  a 
buyer  to  consider  the  system.  If  even  one  of  the  core  requirements  cannot  be 
immediately  met  by  the  LMS,  that  LMS  should  be  eliminated  from  the  list.  For  this 
reason,  only  core  requirements  that  truly  represent  imperative  functionality  should  be 
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incorporated.  All  other  requirements  must  be  considered,  but  the  focus  should  be  on 
the  core  requirements. 

The  result  of  the  review  should  result  in  a  short  list  of  vendors.  Usually,  the 
shorter  the  list,  the  easier  the  final  selection  process  will  be. 

6.  Schedule  Meetings  and  Demos 

After  the  proposal  review  is  complete,  meetings  and  demos  should  be 
scheduled  so  that  the  vendors  can  answer  specific  questions  and  demonstrate  their 
claims  on  the  proposal.  The  review  committee  should  be  representative  of  all 
stakeholders.  A  scenario  should  be  scripted,  and  vendors  should  be  required  to 
demonstrate  their  systems’  capabilities  in  that  specific  user  case.  This  is  crucial  if  an 
organization  is  to  determine  how  flexible  or  compatible  its  environment  is  with  certain 
systems.  Demos  should  be  scheduled  for  2-3  hours  each  and  should  be  completed 
in  a  compressed  time  frame  of  2-3  days  to  facilitate  comparisons.  It  is  also  important 
that  vendors  make  clear  what  part  of  the  functionality  is  included  out-of-the-box  with 
minor  configuration  changes  and  what  part  requires  customization  beyond  the 
quoted  price. 

If  any  of  the  review  team  needs  to  attend  such  a  meeting  virtually,  it  would  be 
good  opportunity  to  utilize  a  vendor’s  distance-learning  solution.  This  will  provide  the 
capability  to  experience  part  of  the  environment  as  the  learning  audience  would. 

The  review  committee  must  question  any  part  of  the  functionality  or 
implementation  that  is  not  clearly  understood.  It  is  important  that  the  account 
representative  is  able  to  explain  functionality  clearly  and  without  ambiguity. 
Additionally,  the  flexibility  of  the  project  plan  should  be  explored.  An  organization 
should  not  be  required  to  adhere  to  processes  that  conflict  with  internal  processes. 

7.  Make  the  Selection 

Finally,  a  selection  can  be  made  after  an  organization  carefully  reviews  and 
discusses  the  impressions  made  by  each  vendor  during  each  meeting.  Selection  of 
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a  LMS  is  a  serious  and  long-term  investment,  so  it  is  important  to  have  complete 
cooperation  among  the  members  of  the  review  committee.  It  is  also  important  to 
create  contingency  plans  in  case  certain  features  that  are  expected  in  the  initial 
implementation  are  not  finished  in  time,  or  other  unexpected  delays  or  problems 
arise. 


The  steps  outlined  above  represent  a  high-level  approach  to  selecting  a  LMS 
to  meet  an  organization’s  learning  requirements.  The  approach  is  flexible,  and  the 
process  cycle  can  be  shortened,  although  usually  at  the  cost  of  quality.  It  is 
important  to  note  that  a  LMS  solution  can  be  a  huge  investment,  and  if  the  selection 
and  implementation  is  not  done  thoroughly,  can  lead  to  enormous  costs  later  in 
additional  efforts  to  meet  requirements. 
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VI.  Pitfalls  for  Selecting  and  Implementing  a  LMS 


There  are  many  things  that  can  go  wrong  when  an  organization  is  selecting 
and  implementing  a  LMS  (Egan,  2002;  Hultin,  2005).  The  following  are  some  of  the 
common  issues  that  organizations  face  when  selecting  and  implementing  a  LMS 
solution: 


■  Failure  to  consider  the  LMS  as  a  mission-critical  system.  As  a  result, 
its  implementation  gets  little  or  no  IT  support. 

■  Failure  to  align  business  objectives  with  the  capabilities  of  the  LMS, 

■  Failure  to  understand  how  a  LMS  interacts  with  other  systems  and 
business  units, 

■  Absence  of  an  organizational  learning  strategy  to  drive  the  selection  of 
the  LMS, 

■  Failure  to  identify  and  engage  all  key  stakeholders, 

■  Failure  to  obtain  upper  management  buy-in  and  support, 

■  Setting  up  unrealistic  expectations, 

■  No  or  ill-defined  user  requirements,  and 

■  Lack  of  internal  skills  to  support  and  configure  the  LMS. 

The  above  issues  are  typical  of  software  implementations  in  general  and  LMS 
implementations  in  particular.  They  represent  some  (but  not  all)  problems  that  can 
be  encountered  during  the  course  of  selecting  and  implementing  a  LMS.  Discussing 
these  issues  will  hopefully  raise  awareness  and  prepare  users  and  vendors  to  plan 
ahead  and  avoid  pitfalls. 
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VII.  Evaluating  Learning  via  Learning  Technologies 


Delivering  training  via  learning  technologies  (i.e.,  LMSs)  does  not  guarantee 
an  organization  a  return  on  investment  or  that  the  learner’s  performance  improved. 
One  of  the  important  issues  facing  LMS  implementation  is  proving  whether  a  learner 
actually  gained  knowledge  and  was  able  to  transfer  the  knowledge  to  the  workplace 
and,  ultimately,  to  improve  performance.  The  evaluation  of  training  is  a  tricky 
subject,  and  numerous  proposals  by  researchers  and  practitioners  were  advanced  to 
measure  it  from  “smile  sheets”  to  “return  on  investment”  (ROI)  with  mixed  results 
(Theis,  2005).  A  widely  used  model  for  evaluating  training  was  proposed  by  Donald 
Kirkpatrick  (1998).  Kirkpatrick’s  model  consists  of  four  levels  of  training  evaluation.  It 
is  used  for  evaluating  training  effectiveness,  whether  that  training  was  conducted  via 
instructor-led  classroom  experiences  or  via  e-learning  experiences.  A  summary  of 


Kirkpatrick’s  four  levels  of  evaluation  follows: 

Level  1 : 

Did  they  like  the  training? 

Level  2: 

Did  they  learn  something  from  the  training? 

Level  3: 

Did  the  training  help  them  do  their  job  better  and  increase 
performance? 

Level  4: 

Did  the  company  or  department  increase  profits  and  customer 
service  as  a  result  of  the  training? 

Phillips  and  Stone  (2002)  adopted  Kirkpatrick’s  four  levels  evaluation  and 
adapted  it  by  adding  a  fifth  level  and  applying  six  different  measurements  for 
calculating  the  effectiveness  of  an  intervention.  Table  1  summarizes  their  model. 
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Level  and  Data  Type 

Focus  of  Data 

Data  Usefulness 

Level  1 :  Reaction  and 
Satisfaction 

Training  program, 
facilitator,  planned  action. 

Documents  reaction  to 
training  by  learners. 

Level  2:  Learning 

Participant  and  support 
mechanisms  to  assist 
learning. 

Measures  knowledge, 
skill  and  attitudes  learned 
from  the  training. 

Level  3:  Job  Application 

Participant,  work-setting, 
and  support  mechanisms 
for  applying  learning. 

Measures  behavioral 
change  on  the  job. 

Level  4:  Business  Impact 

Impact  of  the  training 
process  on  specific 
organizational  outcomes. 

Determines  the  training 
impact  in  improving 
organizational 
performance. 

Level  5:  ROI 

Monetary  benefits  as  a 
result  of  training. 

Evaluates  the  monetary 
value  of  the  business 
impact  of  the  training  as 
compared  to  the  cost  of 
the  training. 

Intangible  Benefits 

The  added  value  of  the 
training  in  non-monetary 
terms. 

Describes  the  intangible 
data  (subjective)  that 
emerge  in  evaluation  of 
business  impact. 

Table  1.  Five-level  Model  for  Evalual 

:ing  Learning 

(adapted  from  Phillips  &  Stone,  2002) 


As  with  most  theories  and  models,  there  are  differences  in  opinions  as  to  how 
accurate  evaluation  can  be.  Kirkpatrick’s  model  and  Phillips  and  Stone’s  model 
above  focus  on  specific  training  experiences.  Additional  researchers  focus  on 
performance  management  that  is  linked  to  corporate  strategies  and  goals  (see,  for 
example,  Kaplan  &  Norton,  1992). 
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VIII.  The  Case  against  Learning  Management 
Systems 


It  has  been  suggested  that  LMSs  in  general  are  the  wrong  starting  point  for 
learning  (Siemens,  2004;  Siemens,  2006): 

Learning  Management  Systems  are  often  viewed  as  a  central  point  of  any 
elearning  or  blended  learning  program.  Although  this  perspective  is  desirable 
from  a  management  and  control  standpoint,  it  is  contrary  to  the  way  in  which 
most  people  learn  today. 

Learning  management  systems  [...]  offer  their  greatest  value  to  the 
organization  by  providing  a  means  to  sequence  content  and  create  a 
manageable  structure  for  instructors/administration  staff.  The  "management" 
aspect  of  LMS'  creates  another  problem:  much  like  we  used  to  measure 
"bums  in  seats"  for  program  success,  we  now  see  statistics  of  "students 
enrolled  in  our  LMS"  and  "number  of  page  views  by  students"  as  an  indication 
of  success/progress.  The  underlying  assumption  is  that  if  we  just  expose 
students  to  the  content,  learning  will  happen  (2004,  p.  1). 

Although  certain  learning  tasks  are  well  suited  for  a  LMS,  learning  itself  is 
different — it  is  not  a  process  to  be  managed.  Learning  is  by  nature  multi-faceted  and 
chaotic  (Conner,  2007).  The  LMSs  of  today  define  the  manner  in  which  learners 
learn.  Their  structured  approach  dictates  the  nature  of  interaction  (instructors- 
learner,  learner-learner,  learner-content).  Organizations  that  now  lock  into  learning 
management  systems  will  be  able  to  successfully  deliver  and  manage  courses;  they 
won't,  however,  be  conducting  effective  learning.  The  idea  of  acquiring  a  large, 
centralized,  mono-cultural  software  tool  that  can  do  it  all  is  flawed  (Parkin,  2004). 

The  more  feature-rich  an  individual  tool  becomes,  the  more  it  loses  its  usefulness  to 
the  average  user.  What  is  needed  is  a  diversity  of  tools  and  choices,  such  as:  blogs, 
wikis,  content  management  systems,  social  networking  tools,  collaborative  spaces, 
and  the  use  of  emerging  "connection-enabling"  protocols  like  RSS — that  provide 
connected  specialization,  modularization,  and  decentralization  and  that  place  the 
learner  at  the  center  of  learning  and  allow  him/her  to  explore  various  areas  and 
directions  of  personal  interest. 
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As  Siemens  (2004)  states: 


Unfortunately,  beginning  learning  with  a  LMS  is  often  a  matter  of  wrong  tool 
for  wrong  purposes  (which  results  in  failed  elearning  implementations, 
ineffective  learning,  and  unnecessary  expenses).  Implementing  a  LMS  as  part 
of  a  holistic  learning  environment  gives  the  end  user  flexibility  and  control  to 
move  in  various  paths  (driven  by  learning  needs,  not  by  LMS  design),  (p.  1) 
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IX.  What  Does  an  Ideal  Learning  Environment 
Look  Like? 


As  discussed  in  the  previous  section,  an  ideal  learning  environment  must 
ensure  that  an  organization  is  providing  effective  learning,  not  just  delivering  content. 
Specifically,  a  learning  environment  should  be  able  to  (Masie,  2006): 

1 .  Personalize  invitations  to  learn  and  target  them  for  each  learner  based 
on  current  and  future  requirements,  performance  goals,  and  critical 
events. 

2.  Support  different  stages  of  learning:  initial  learning,  continued  learning, 
remedial  learning,  upgrade  learning,  and  transferred  learning  to  new 
technology. 

3.  Support  different  types  of  learning:  behavioral,  cognitive,  and 
sociocultural. 

4.  Create  an  executive  “dashboard”  for  each  manager  that  highlights,  in 
real  time,  how  his/her  subordinates  and  teams  are  learning. 

5.  As  rules  and  regulations  change,  inform  employees  and  offer  updated 
learning  modules. 

6.  Integrate  the  organization’s  social  networking  and  facilitate  informal 
learning. 

7.  Understand  the  learning  styles  of  different  learners  and  offer  learning 
options  appropriate  for  each  employee,  content  domain  and  situation. 

8.  Incorporate  new  and  emerging  content  models  such  as  podcasting, 
wikis,  blogs,  social  networking  tools,  and  collaborative  spaces. 

9.  Become  part  of  day-to-day  online  automated  tools,  such  as  browsers. 

10.  Become  integrated  with  the  organization’s  core  mission  and  objectives 
and  adapt  accordingly  to  any  changes. 

1 1 .  Work  with  a  wider  range  of  devices,  such  as  PDAs  and  mobile  devices. 

12.  Provide  measured,  just-in-time  learning. 

13.  Access  an  increasing  number  of  reusable  content  both  inside  and 
outside  the  organization. 
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14.  Integrate  with  other  systems  that  share  common  objectives,  such  as 
Knowledge  Management  and  Document  Management  Systems. 

1 5.  Keep  track  of  and  share  employees’  assessment  of  learning  modules. 

16.  Become  part  of  the  organization’s  crisis  management  plan  by 
delivering  extremely  rapid  content. 

Although  many  of  these  features  are  supported  in  varying  degrees  across  the 
current  offerings  of  LMSs,  no  tool  currently  exists  that  comes  close  to  implementing 
the  feature  list  of  an  ideal  LMS.  This  situation  creates  a  challenge  for  organizations 
trying  to  identify  an  approach  for  managing  learning.  Two  options  exist  for 
organizations  implementing  a  learning  program:  1)  either  work  with  LMS  vendors  to 
restructure  their  systems  to  incorporate  many  of  the  features  missing  from  their 
respective  systems,  or  2)  abandon  the  adoption  of  a  single  LMS  altogether  and 
develop  an  alternative  approach  based  on  a  toolbox  of  decentralized,  modularized, 
learner-centric,  piece-it-together  tools,  as  described  in  the  previous  section. 
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X.  Learning  Management  Systems  Trends 


A  survey  of  the  CEOs  of  the  top  learning  management  systems  vendors 
indicates  the  following  trends  in  the  next  few  years  (Masie,  2007a;  2007b;  2007c; 
2007d;  2007e;  2007f): 

1 .  Learning  management  systems  continue  to  become  a  robust,  stable, 
and  tested  platform. 

2.  Multiple  learning  management  systems  in  organizations  are  being 
consolidated  into  one  enterprise  system. 

3.  Learning  management  systems  are  moving  away  from  a  focus  on 
compliance  training  to  strategic,  mission-critical,  innovative 
applications — such  as  just-in-time  learning. 

4.  As  a  result,  LMS  projects  are  being  viewed  and  deployed  as  mission- 
critical  systems,  and  are  thus  getting  higher  visibility  and  support  from 
upper-level  management. 

5.  Increasingly,  learning  management  systems  are  being  integrated  with 
enterprise  systems  such  as  Human  Resources  (HR),  Content 
Management  Systems  (CM),  Enterprise  Resource  Management 
(ERP),  Financial,  and  other  enterprise  systems. 

6.  LMSs  are  becoming  particularly  HR-centric  and  are  moving  towards 
driving  talent  and  performance  management. 

7.  Learning  management  systems  are  increasingly  being  accessed, 
together  with  other  enterprise  systems,  through  customized  and 
personalized  corporate  portals. 

8.  More  emphasis  is  being  given  to  the  user  experience  and  “ease  of  use” 
of  a  LMS. 

9.  Stronger  emphasis  is  given  to  informal  learning  by  incorporating  wikis, 
blogs,  PDFs,  and  knowledge  and  social  network  elements  that  connect 
people  through  interests,  competencies,  skills,  and  expertise. 

10.  The  increased  use  of  consolidated  searches  is  exposing  the  end  user 
to  learning  in  a  more  blended  fashion.  These  capabilities  allow  users  to 
access  both  formal  and  informal  content  using  a  unified  metaphor. 

1 1 .  Increasingly,  LMS  vendors  and  customers  are  adopting  standards 
such  as  SCORM  and  LETSI  for  content  and  integration. 
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12.  Recently,  vendors  have  moved  towards  Web  Services  and  Multi-tenant 
Architectures — in  which  a  LMS  is  used  as  a  service  rather  than  being 
acquired  and  run  in  house.  This  approach  will  lead  to  greater 
integration  between  e-learning  and  other  enterprise  systems. 

The  field  of  learning  systems  is  continuously  evolving.  It  is  prudent  for 
organizations  to  anticipate,  influence  and  understand  future  trends  of  the  industry 
and  to  match  their  current  and  future  requirements  with  these  trends. 
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XI 


Conclusions 


This  research  addressed  the  suitability  and  use  of  learning  management 
systems  as  high-level,  strategic  solution  for  planning,  delivering,  and  managing 
learning  within  large  organizations,  such  as  the  Navy.  The  research  addressed  many 
issues  related  to  the  selection,  implementation,  and  use  of  learning  management 
systems — such  as  the  current  state  of  the  learning  management  systems  industry, 
the  essential  functional  and  technical  requirements  of  an  effective  learning 
management  system,  the  process  and  pitfalls  for  selecting,  implementing,  and 
evaluating  a  learning  management  system,  the  case  against  using  a  learning 
management  systems,  and  future  trends  and  directions. 

Although  a  learning  management  system  is  extremely  useful  in  delivering  and 
managing  a  wide  variety  of  structured  content,  the  notion  of  “managing  learning” 
conflicts  with  how  people  are  actually  learning  today.  In  many  situations,  learners 
require  a  learning  environment  that  allows  for  a  very  quick  learning-structure 
creation  and  breakdown  and  that  is  continuously  available.  A  learning  management 
system  has  a  long  creation/breakdown  process,  and  once  the  learning  structure  has 
been  broken  down  (i.e.,  end  of  course),  it  is  no  longer  accessible  to  learners.  In  such 
situations,  rather  than  adopting  a  large,  centralized,  closed,  mono-cultural  LMS  that 
dictates  the  nature  of  interaction  between  learners,  instructors,  and  content,  we 
recommend  the  Navy  adopt  an  integrated  learning  environment  (ILE)  that  consists  of 
a  set  of  diverse  tools  that  provide  rapid  creation  and  breakdown  through  connected 
specialization,  modularization,  and  decentralization.  Such  a  structure  places  the 
learner  at  the  center  of  learning  and  allows  him/her  to  explore  various  areas  and 
directions  of  personal  interest. 

Regardless  which  approach  is  adopted,  organizations  must  first  have  a  clear 
understanding  of  the  learning  environment,  develop  a  clear  learning  strategy  and 
architecture,  and  prioritize  requirements  before  looking  for  a  learning  system. 
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